Cost-cognizant Test Case Prioritization

Test case prioritization techniques schedule test cases for regression testing in an order that increases their ability to meet some performance goal. One performance goal, rate of fault detection, measures how quickly faults are detected within the testing process. Previous work has provided a metric, APFD, for measuring rate of fault detection, and techniques for prioritizing test cases in order to improve APFD. This metric and these techniques, however, assume that all test case and fault costs are uniform. In practice, test case and fault costs can vary, and in such cases the previous APFD metric and techniques designed to improve APFD can be unsatisfactory. This paper presents a new metric for assessing the rate of fault detection of prioritized test cases, APFDC , that incorporates varying test case and fault costs. The paper also describes adjustments to previous prioritization techniques that allow them, too, to be “cognizant” of these varying costs. These techniques enable practitioners to perform a new type of prioritization: cost-cognizant test case prioritization. Finally, the results of a formative case study are presented. This study was designed to investigate the cost-cognizant metric and techniques and how they compare to their non-cost-cognizant counterparts. The study’s results provide insights regarding the use of cost-cognizant test case prioritization in a variety of real-world settings.